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ABSTRACT 

A series of occupational competency tests 
representing all seven vocational education curriculum areas were 
developed, field tested, and validated. Seventeen occupations were 
selected for competency test development: agricultural chemicals 
applications technician, farm equipment mechanic, computer operator, 
word processing specialist, apparel sales, fabric sales, grocery 
clerk, hotel (motel) front office, dental assistant, physical 
therapist assistant, custom sewing, restaurant service (waiter, 
waitress, and cashier), electronics technician, water treatment 
technician, wastewater treatment technician/ carpenter, and diesel; 
mechanic. Identification of competencies involved development of a 
task inventory and task verification through .interviews in the field. 
Test items were reviewed by experts representing vocational educators 
and employers, and the tests were' revised and pilot tested. Over 3500 
students in more than 150 sites in 37 states participated in the 
field testing. Contents of the final test packages were a job 
information (paper-and-pencil) test, a complete set of hands-on 
performance tests, a Work Habits inventory (a teaching and counseling 
tool in job survival skills), and an Examiner's Manual. Test 
reliability and validity were also investigated, and a wide range of 
dissemination activities were undertaken to stimulate test usage and 
additional test development. Appendixes include a copy of the Work 
Habits Inventory. (TLB) 
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ABSTRACT - 

Between October' 1979 and December 1982, the American Institutes for 
Research was engaged in a y nationwide effort to develop, field test, and 
validate 17 occupational ^competency tests under the sponsorship of the 
U. S. Department of Education. Other goals of the project were to promote 
the acceptance of student competency testing in vocational education and 
to help stimulate the continuing development of occupational competency 
tests. 

After extensive interaction with many leaders in vocational education 
the following occupations were selected for competency test development: 

Agriculture - Agricultural Chemicals Applications Technician, Farm 
Equipment Mechanic 
s Business and Office - Computer Operator ,^ Word Processing Specialist 
Distributive Education - Apparel Sales,, Fabric Sales, Grocery Clerk, 

Hotel (Motel) Front Office 
Health - Dental Assistant, Physical Therapist Assistant 
Home Economics - Custom Sewing, Restaurant Service (Waiter, 

Waitress, Cashier) 
Technical - Electronics Technician, Water Treatment Technician, 

Wastewater Treatment Technician 
Trade and Industry - Carpenter, Diesel Mechanic 

The competency tests are intended to serve two major purposes: (1) tc 
help teachers and administrators of secondary, and postsecondary vocational 
education programs evaluate and improve specific areas of their vocational 
programs, and (2) to provide an objective basis for informing students', 
teachers, and prospective employers about the progress made by students in 
acquiring specific, job-related competencies. Employers may also find the 
tests useful for helping in selecting new employees and assessing objec- 
tively the training needs of present employees. 

Each test package includes the following items: 

• paper-and-pencil test: two parts, each part requiring no 
longer than one class 'period to administer 

• — a-eom pie t-e— s e ^of- -the- u hand s- on— per-f-or raanee-t-e s t-s— £ or 

that occupation, each test containing examiner and examinee 
instructions, appropriate test props, and a test record 
sheet; the number of performance tests in each package 
ranges from 4 to 13 

• Work Habits Inventory, for use as a teaching and 
counseling tool in job survival, skills 

• an Examiner's Manual, including directions for test 
administration, a summary of how the tests were 
developed, technical data on test reliability and 
validity, and scoring keys for the tests and the 
Work Habits Inventory 
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The results" of the f ield .testing and validation have shown that: 



All of the competency tests demonstrated good to excellent 
reliabilities. 

The employer appraisal of test relevance — our primary measure 
of the relevance of the tests to job requirements — indicated 
that the content of each of the tests was rated important to 
the* job. 7 

For most of the Job Information Tests, where adequate data were 
available, the employees with substantial amounts of experience 
scored higher on the average, than those with less experience, 
who in turn scored higher than individuals in training • Those 
with neither training nor experience scored the lowest • 

To stimulate increased usage of student competency measures in voca- 
tional .education and the development of additional measures, a wide range 
of dissemination activities was undertaken, including the preparation of, 
a report on the state of the art of competency measurement in vocational 
education and four instructipnal manuals covering the effective use of 
occupational competency measures, their development, and their validation. * 




INTRODUCTION 

In the fall of 1979, the American InsCituCes for Research (AIR) 
responded successfully Co a requesc f or ^proposals (RFP) from whac now the 
Office of Vocacional and Adulc Educacion of the U.S. QeparCment* of Education. 
This RFP, issued largely in response .to requescs from- Scate DirecCors of 
Vocacional Education, called for Che design, development, field CesC, and 
dissemination of a series of occupational^compecency measures representing 
all seven vocacional educacion curriculum areas. 1 ' 



The CompeCency tescs are incended co serve two major purposes:- (1) to 
help ceachers and administrators <ff secondary and post secondary vocacional 4- 
educacion programs evaluate and improve "pecific areas of their vocacional 
programs, and (2) Co provide an objective basis for informing studencs, 
Ceadhers, and prospective employers about Che progress made by scudent-s in 
acquiring specific, job-related competencies. Employers may also find Che 
Cests useful for helping in selecting new employees and assessing objeccively 
Che craining needs of presenc employees. 

The specific objecCives of che projecC were as follows: * 

• Co develop compeCency cests in selecCed occupations 
represencing all seven vocational curriculum areas 

• Co establish their usefulness through extensive field 
Cescing and validation * ; 

• Co promoce cheir acceptance and use in vocational educa- 
cion programs 

• tp design and help implemenc a program for continuing 
occupacional competency CesC developmenc on a self- 
supporcing basis 

To help in the planning and implementacion of che projecC, AIR was 
assisced by *two advisory groups: che project National Policy Council and 
che National Sub^ecc Matter Panel* The 18-member NaCional Policy Council 
advised on overall project design, including mechods for field testing and 
validation and strategies for dissemination* 
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The 14-member National Subject Matter Panel advised project staff on* 
criteria for occupational selection and recommended technical reviewers of 
the test items*/ Wherever appropriate, subject matter panelists also served 
as test reviewers* 
* 

The members of each advisory group are listed in Appendix A. 

The purpose of this report is to summarize the major activities under- 
taken during the course of t£e Vocational Competency Measures (VCM) project. 
■•We should emphasise, however, that the primary justification for the project 
is the Series of 17 occupational competency test packages produced under the 
\ contract • Contents of these test packages are described in a later section. 
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SELECTION OF OCCUPATIONS FOR TEST 
DEVELOPMENT 

Our process for selecting occupations reflected several general factors. 
First was our concern that the areas selected should lead to tests that would 
be accepted by and useful to the vocational educators in all states, both as 
measures of student competency and as stimuli for curriculum and instruc- 
tional development. Second were the recommendations made by our National 
Policy Council, our National Subject 'Matter Panel, State Directors of Voca- 
tional Education or their representatives, and other leaders in vocational 
education The resulting criteria were a composite of those perspectives. 



* Our initial goals were to select: 

• two occupations representing each of the seven vocational 
curriculum areas: agriculture, business and office, dis- 
tributive education, health, home economics, technical, 
trade and industry 1 

• occupations representing differing principal segments within 
a given area 

• occupations in which opportunity for employment is favorable 
and for which there are ample enrollments, or occupations in 
emerging areas that may not yet .have high enrollments but 
where the trend is upward 

• occupations in which "adequate" competency tests (including 
performance components) are not available to vocational edu- 
cators or currently under development 

• occupations where vocational training is necessary (as 
opposed to unskilled entry-level occupations in which there 
is high turnover of temporary help) 

• occupations wh$re there is a good consistency in the content 
across different geographical regions (to assure national 
rather than strictly regional applicability) 

• occm>ations that, in the aggregate, would represent a range 
of diverse models* to guide future developers of tests beyond 
the original tests that AIR would prepare 



This was expanded to 17 occupations as will be described later. 



The AIR project staff communicated extensively with the project f s 
National Policy Council and with its National Subject Matter Panal. Both 
groups gave us a vote of confidence that our procedures for selecting occu- 
pations were reasonable and fair» Following these criteria, project staff 
reviewed many possible occupations that seemed worthy of consideration for 
the development oi competency tests that would be applicable across the 
nation, including cognitive, affective, and performance dimensions. 

As- we began to focus on specific occupational choices in each of the 
seven areas, we communicated by telephone with Panel members, with selected 
Council members,- with authorities in the particular occupations, with 
resource centers, such as the Vocational-Technical Education Consortium of 
the States (V-TECS) and the Mid-America Vocational Curriculum Consortium 
(MAVCC), and with selected state leaders in vocational education. In the 
course of this search, we found a number of areas that, because of overlap, 
lack of universal appeal and consistency, or other reasons, were then dropped 
from consideration. 

In order to rank occupations in order of priority for possible compe- 
tency test development, it was clearly advisable that a comparative analysis 
be undertaken of vocational education enrollment, labor trends, employee turn- 
'over, and vocational complexity levels. Using appropriate references from 
the Bureau ot the Census, the Bureau of Labor Statistics, and the then U.S. 
Office of Education, a draft table was constructed containing this informa- 
tion for selected high-enrollment programs in each of the seven vocational 
areas* Also included in each table was preliminary information on the cur- 
rent and projected availability of competency measures for specific occupa- 
tions* The table for each area was sent to the two Panelists representing 
that specialty, along with an outline of our proposed strategy for occupa- 
tional selection. Panel members were asked to appraise our strategy, to 
recommend specific occupations for test development, and to provide addi- 
tional information and leads to other sources". 

In order not to delay che project schedule, it was necessary to select 
three occupations for initial test development. Prior to our final selection 
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of the first three occupations, we conferred with the current and past presi 
dents of the National Association of State Directors cf Vocational Education 
and obtained the approval of the Project Officer. After this initial selec- 
tion had .been completed, a letter was sent to all Sr.ate and Territorial 
Directors of Vocational Education informing them of the first three occupa- 
tions selected for test development (and the criteria for their selection) 
and requesting their recommendations for the remaining occupations. 
Responses were received from 19 states and? outlying territories. 

Selection of the remaining occupations proceeded in increments, with 
the final increment being submitted and approved by the Project Officer in 
June 1986. The procedures for selection were„ similar to those followed 
earlier and consisted of analysis of enrollments and labor projections; 
identification of available performance-centered tests and task inventories; 
and review of nominations from the National Policy Council members; National 
Subject Matter Panelists, and State Directors. Discussions with program 
specialists in the Office of Vocational and Adult Education, with leaders in 
professional associations and related governmental agencies, and current and 
past presidents of the National Association of State Directors of Vocational 
Education also helped to clarify the priorities among alternative areas. 

In accordance with our objectives, a total of 14 occupations were 1 
selected for test development. Based on experience gained during our data 
collection, two occupations were subdivided to reflect more closely actual 
job content, resulting in a total of 17 occupational test packages. The 
final list cf occupations selected for test development is shown in Table !• 



TABLE 1 



Occupational Areas Covered in 
the AIR Vocational Competency Tests 



• AGRICULTURE 

Agricultural Chemicals 
AppSications Technician 

'Farm Equipment Mechanic 

• BUSINESS AND OFFICE 

Computer Operator 

Word Processing 
Specialist 

• DISTRIBUTIVE EDUCATION 

Apparel Sales 

Fabric Sales 0 

Grocery* Clerk 

Hotel (Motel) Front 
Office 



• HOME ECONOMicS 

Custom Sewing 

Restaurant Service 
(Waiter, Waitress, 
Cashier) 

• TECHNICAL 

Electronics Technician 

Water Treatment 
Technician 

Wastewater Treatment 
Technician 

• TRADE AND INDUSTRY 

Carpenter * 
Diesel Mechanic 



• HEALTH 

Dental Assistant 

Physical Therapist 
Assistant 
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IDENTIFICATION OF COMPETENCY REQUIREMENTS 

Identifying Che competencies in each of the selected occupations 
consisted of five stages: 

• Gathering background data on selected occupations ^ ^ 

• Developing interview instruments and checklists 

• Conducting interviews and gathering task verification infor- 
mation in the field- 

• Analyzing obtained information 

• Summarizing and prioritizing findings for test development 

For each occupation selected, the above stages were followed in identifying 
required job competencies. 

Gathering Background Data On Selected Occupations 

The first step was to gather existing inventories of the tasks per- 
formed in these occupations. All available avenues (published sources, 
governmental sources, military sources, personal contacts) were pursued, a^nd 
a wide variety of inventories were obtained. These inventories were reviewed 
and a compilation of all nonredundant skills and knowledge was then developed 
for each occupation. For the initial 14 occupations, this acquisition and 
compilation task resulted in a rich foundation of information about compe- 
tencies* 

Developing Interview Instruments and Checklists 

Each listing was then reformatted for use as a structured checklist 
that could be self-administered. The format used for all the task inventory 
surveys was kept similar for CvOnsistency in use" and analysis. 

A pilot test of the checklist was conducted at a sophisticated elec- 
tronics firm and at an auto repair shop to determine its usability under 
widely varying circumstances and in diverse areas* 
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Each respondent was asked to provide information on task frequency, 
importance, and whether the task was learned on the job or before being 
hired. An excerpt from one of the checklists is shown below. 

Frequency Importance Learned 

££ E WEEKLY MONTHLY HI6H MODKRATE LOW BEFORE AFTER 

1. Maintain hand tools. 

2. Maintain pov*r tools. 

3. Maintain surveying instru- 

•ants and equipment. 

* - 

The checklist yielded information on the frequency and significance of the 
competencies as well as some idea of whether each could reasonably be 
assessed in a pre-employment school setting. Space was also provided for 
Respondents to "write in" and rate additional tasks not covered in the task 
inventory survey. 



The respondents were also asked to name personal qualities that they 
felt were important for the job in three general areas. These were: 

- personal characteristics, such as -being dependable * 

- work habits, such as starting work on time 

- interpersonal, relations, such as accepting supervisor's 
suggestions ■ 

Finally, the respondents were asked whether substantial parts of the job - 
could be performed .by persons with the following types of handicaps: 

- orthopedic handicaps 

- deaf or hard of hearing 

- - blind or visually impaired 
-mentally retarded 

When respondents completed the checklist they were asked to go over the 
list and circle up to 10 of the items that they v felt were most critical to 



the job. The circled items were used as a check on the arranging .of compe- 
tencies in order of imp< 
into performance tests. 



tencies in order of importance and the v selection of those to be developed 



A general interview guide was also prepared for use in conjunction with 
the task inventory checklist. This guide gave suggestions for open-ended 
prompts f or* use by the AIR representative. 



Conducting Interviews and Gathering Task Verification Information 
in the Field 

Once the task inventory and interview forms were developed, arrangements 
were made to Interview supervisors and workers throughout the country. For 
all the occupations selected, a conscious effort was made to get a distribu- 
tion of dif f erfent-sized companies and businesses to eliminate any possible 
bias that could exist due to size. Diversity was also sought in the type of 
business that utilized workers in a particular occupation. It was feit that, 
for example, a diesel mechanic repairing tractor engines may have a somewhat 
different perception of what tasks are important or most often done than a 

diesel mechanic repairing truck engines. Similarly, diversity was sought 
♦ 

along geographic lines in an effort to avoid regional v bias. For example, 
farm equipment differs regionally, yet we were interested in identifying 
competencies that were generaliza ble_acxoss^-he-^countTy^ " r 

Names and locations of possible firms were received from: 

- National Policy Council members and National Subject 
Matter panelists 

- State Directors of Vocational Education 

- professional organizations 

In scheduling site visits, efforts were made to cluster them for the various 
occupations in order to reduce travel costs as much as possible. 

Altogether, our task verification phase involved interviews, in 27 states 
and the District of Columbia. Table 2 shows the number of interviewees and 
states involved in the task, verification of each .occupation selected for 
test, development r^^lR representatives interviewed job incumbents and super- 
visors separately. Throughout, the interviewees were encouraged to expand 
on any additional competency areas' that' were especially important for per- 
formance appraisal. 

Analyzing Obtained Information 

Both quantitative and qualitative procedures were followed in analysis 
to assure maximum utilization of all the information that was collected. The 
general information obtained through interviews was qualitatively analyzed 
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TABLE 2, 

Number of Interviews and States Involved in the 
Gathering of Task Verification Information 

Test Nam6 No. of Interviews States Involved 



Agricultural Chemicals 19 CA, GA, MO, NY, UT 
Applications Technician 

Farm Equipment Mechanic 25 > CA, NY, OH, OK, WI 

Computer Operator 24 v CA, IA, NY, OK 

<# 

Word Processing Specialist -> 12 CA, IN, TX 

Grocery Clerk 18 AL, AZ, CA, DC, VA 

Hotel (Motel) Front Office 13 CA, MA, PA, SC 

-Dental-Assistant - . — -20 CA, NJ ^JM, NY, OK 



Physical Therapist 12 CA, IN, TX 
Assistant 

Fashion/Fabric Sales U CA, GA, IL, MO, RI 
and Sewing 

Restaurant Service (Waiter, 14 AL, CA, IL, MA, NY, 

Waitress, Cashier) PA, SC 

Electronic Technician. 16 CA, FL, MA, MD, TN 

Water /Wastewater 13 CA, FL, MA 
Technician 

Carpenter 14 CA, KY, MA; NM, NV, 

SC 

Diesel Mechanic 12 AZ, CA, MD, MN 
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by test team leaders. This analysis helped to focus on topics that might be 
relevant to performance assessment, such as certain kinds of equipment used 
for particular tasks, criteria for judging quality performance, etc. 

For the listing of items oi; the task inventory survey, the analysis 
procedure was more quantitative. For each task inventory item (typically 
around 100. per inventor/), the ratings were separately summed and averaged 
for Supervisors and job incumbents. The total score was then weighted to 
give responses from supervisors additional importance in the final weighting. 
(This is realistic both because of their greater experience and their func- 
tion as evaluators of "nevr-hire" vocational trainees during their early 
months of employment.) 

Inventory items where the weighted-average score vindicated that the 

task was considered important, frequently performed, or both were then 
selected as areas for test item development. Excluded from test item devel- 
opment were those tasks considered less important or considered important 
only in a limited geographical, region. On completion of the analysis, the 
summaries of the findings were then given to the test team leaders to. begin 
developing the competency tests. 

The final task inventories for each of the occupations selected for 
test development includes all tasks that were found to have even moderate 
importance in our task inventory survey or to have been performed at least 
on a morfthly basis — not just the tasks that are covered in the AIR competency 
measures. These lists are being submitted to the East Central Network for 
Curriculum Coordination, a federally-sponsored national curriculum network 
located at Sangamon State University., Springfield, Illinois for incluoion in 
their data bank of task lists. The task lists will also be submitted to the 
Educational Resources Information Center (ERIC) of the U.S. Department of 
Education. 
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This stage encompassed tturee major activities: 
!• Item development/and tryout 



3. 



2. 



field testin: 



pilot testing 




Item Development and Tryout 



For each test, AIR staff worked with two or more technical experts in 
preparing the final test outline and the individual test items. To further 
ensure that the tests reflect real job demands and, at the same time., are 
sensitive to the problems faced by vocational educators, each test was 
reviewed by three or more experts representing vocational educators, as well 
as employers not associated with the development of the test items. Techni- 
cal reviews of each test were solicited from at least two regions of the 
country. Following test review, each test was revised. 

Pilot Testing 

Pilot testing consisted of tryout s of each test with a small group of 
students typically in two .locations. Appendix B listo the schools partici- 
pating in the pilot testing. We observed the students taking the test and 
then interviewed them, as well as their instructors, primarily to check on 
our administration procedures and our instructions and to get a preliminary 
estimate of time requirements -of the tests. 

« 

Field Testing 

The competency measures were then field tested in vocational education 
programs across the country. Two tryout forms of each test were developed 
for this purpose. Altogether, over 3,500 students in more than 150 sites., it; 
37 states participated in this field testing. Included in 'this total were 
some 80 Air Force students at two technical training centers — Sheppard and 
Keesler* Air Force bases — and a small number of students at several Navy and 
Marine bases. 
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For purposes of field test planning and coordination, it was conven-, 
ienc to use a geographical grouping of states already in existence — that of 
the National Network for Curriculum Coordination in Vocational and Technical 
Education (the NNCCVTE), as shown in Figure 1. Efforts were made <to have 
each measure field tested in as many of the six regions as possible. 
Table 3 shows the number of schools, number of students, and regions repre- 
sented in che field testing of each test. ^ 

The field tests were used primarily to collect item statistics -for 
reducing the length of the tests and to a lesser extent for modifying items. 
In addition, the field tests provided one basis for estimating test relia- 
bilities. Specifically for each test, the following kinds of item analysis 
data were obtained: 

1. The ;nean score of the examinees reaching the item, on the 
section of the teat that contained the item 

2. 'The corresponding standard deviation 

3. For e ach response option of each item: 

a. the number of examinees selecting the option 

b. the proportion of examinees selecting the op^.io'n 

c. the mean score of those examinees on the corresponding 
section of the test 

d. the corresponding standard deviation 

e. two measures of the extent to which selecting that par- 
ticular response was related to score on the test section 

(1) the point biserial correlation between selecting the 
option and score on the/ test section, and 

(2) the corresponding Brogden-Clemans^ correlation 

• m (These measures are sometimes called indexes of item-test 

homogeneity, of internal consistency.) 



2 Brogden, H. E. A new coefficient: Application to biserial correlation 
and to estimation of selection efficiency. Psychometrika , 1949, 14(3), 
169-182. 

Cleraans, W. V. An index of item-criterion relationship • Educational and 
Psychological Measurement , 1958, 18(1), 167-172. 



' TABLE 3 

Field -Test Participants by School and 'Region ' 



PARTICIPATING IN FIELD TEST 


REGIONS 


TEST 


^SCHOOLS 


#§VtJDENTS 


1 


2 


3 


4 


5 


6 


AGRICULTURAL CHEMICALS 
APPLICATIONS TECHNICIAN 


12 


217 ■ 






x; 


X 


X 




FARM EQUIPMENT 
MECHANIC 


9 


191 


X 


X 






X 


X 


COMPUTER OPERATOR 


9 


203 

* 


X 


X. 


X* 




X 


X 


WORD PROCESSING 
SPECIALIST 




262 


IX 


X 


X 


X 


X 


X 


APPAREL SALES 


7 


* 138 

t. 


X 




X" 


X 




X 


FABRIC SALES 


7 


138 


X 




X 


X 




X 


GROCERY CLERK 


8 


167 


X 




X 


X 


X 


X 


HOTEL <MOTEU FRONT 
OFFICE 


13 


298 


X 


X 


X 


X 


X 


X 


DENTAL' ASSISTANT 


16 


379 


X 


X 




X 


X 


X 


PHYSICAL THERAPIST 
ASSISTANT 


14 


237 


X 


X 


X 


X 


X 


X 


CUSTOM SEWING 


7 


138 


X 




X 


X 




X 


RESTAURANT. SERVICE 
(WAITER* WAITRESS. CASHER) 


12 


238 


X 


X 


X 


X 


X 


X 


ELECTRONICS TECHNICIAN 


16 


403 


X 




X 


X 


X 


X 


WATER TREATMENT * 
TECHNICIAN 


12 


330 ' 


X 


X 


X 


X 


X 




WASTEWATER ^TREATMENT 
TECHNICIAN . 


12 


330 


X 


X 


X 


X 


X 




CARPENTER 


12 


302 


X 




X 


X 


X 


X 


DIESEL MECHANIC 


8 


186 


X 




X 


X 


X 


X 




In interpreting the data resulting from the Item analysis, it was con- 
sidered important chat the Items selected for the final form represent a 
good spread of Ltem difficulties and chat the correlations between item 
response and total test section score be positive and substantial for the' 
right answer and either negative or close to zero for each distractor, and 
also for item omissiont At least as important was the distribution of item 
content, which had to be such that the final form of the test would conform 

to the same specif ications 'as the tryout forms. 

© 

Contents of the gjnal Test Packages 

Each test package includes the following: 

• *job information (paper-md-pencil) test; two parts, with 

each part requiring no longer than one class period to 
administer? * 

• a complete set of the "hands-on" performance tests for 
that occupation; each test containing examiner and exam- 
inee instructions, appropriate test props, and a test 
record sheet (The number of performance tests in each 
package ranges from 4 to 130 « 

• Work Habits Inventory 

• an Examiner 1 s Manual, including d* actions for test ■ 

< administration, a summary of how the tests were developed, 
technical data including test reliability and validation 
information, and the scoring keys for the tests and the 
Work Habits Inventory 

To illustrate what a typical test package looks like, the Computer Operator 
test will be described. The Work Habits Inventory will be discussed in a 
later" section. 

The Job Information Test * Each test is organized into two parts, each 

part taking about 45 minutes or less* We strongly encourage administering 

both parts to each student, particularly when using the test for making 

decisions about an individual student at* the completion of training, or when 

using the test to identify areas wherein individual needs further training. 

On the other hand, there may be occasions when only one part of the test 

need be administered to any one individual, for example, when the test is 

3 

used as an overall progress measure, or as a program evaluation tool. 



For two of the tests, Word Processing Specialist and Water Treatment 
Technician, both parts must be taken* 
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The job information test is organized by major topic or section. For 
purposes of illustration, the organization of the Computer Operator Job 
Information Test is shown below: 

* 

Section -A. General Concepts 

Section B. Storage Media 

Sectio.n C. Peripherals 

Section D. Routine Operations 

Section E. Troubleshooting 

Section F. Auxiliary Equipment 

Section G. The Computer Operator's Job 

All of the items in the job information test are multiple choice, with the 
great majority of the items having five choices. k ■ 

0 

Performance tests. In the Computer Operator Test package there are 
four- performance testaC tfhat require the examinee to actually carry>out some 
cf the entry-level activities expected of a computer operator. They range 
in time required from an estimated 2 minutes to 65' minutes. The tests 
require only the equipment and materials normally available in a school ' 
offering computer operator training or at hpy business or other 'organization 
employing computer operators. Together, the tests cover a range of ^ 
activities commonly performed by computer operators in entry-level jobs; 
however, it is not. expected that all examinees must necessarily be tested on 
the ^entire set. * . * 4 

\ 

\ The decision as to what performance tests to administer will depend on 
the uses that will be made of the test scores. It is suggested that this 
decision be made in conjunction with the appropriate school administrators 
and members of employer advisory committee, after a careful review of the / 
entire set of performance tests. 

All performance tests should be monitored by qualified instructors of 
computer operator programs, who record their appraisals of task performance 
on specially designed Performance Test Record Sheets. In Figure 2 is shown 
fhe t£st record sheet for one of the sub-tests of the Computer Operator 
/ test, operate card reader and console," which is a part of the major task 
entitled "Job Stream: card-to-tape, tape-to-di$k, sort-on-disk, disk-to- 
printer* H 

i ' 18 . 

1 ■ 28 



COMPUTER OPERATOR TEST 
Performance Test 3e: Operate Card Reader and Console 

Performance Test Record Sheet 3e 



„ Examinee 



Examiner 



Date 



School/Employ er_ 



Otolith Day Year 
Start Time 



Observe the examinee's performance and record observations for each of 1 
hAintj. After test is completed, check the items listed under Outcomes. 


the tai 
fc* 


sks 


CARD READER OPERATION 

1. Fans deck before loading 

2. Joggles deck immediately before loading 2. 

3. Uses non-process runout appropriately 3. 
A. Presses end-of-file key at beginning 4. 
5. Invalid card: 

a. Recognizes card is invalid , 5 a « 

b. Makes replacement card according to instructions 
(i.e., follows interpretation) 5b, 

c. Sight-checks replacement card . 5c. 


Yes 


No 






































* 






* 


CONSOLE 

6. Responds promptly 'and correctly to console messages 6. 
, 7. Uses reference manual as necessary 7. 
8. States what job step is executing 

a. First query **a. 
! b. Second query 8b * 


























• 

OUTCOME** 

9. Replacement card is correct (i.e., free of punching 

errors) ^ # 
EXAMINER: Sight-check the replacement card against- the 
invalid card. 









Note to Examiner ; 

"invalid card contains punches 1, 5, 7, 9 in Cofuian 4< 
Replacement card should be identical except that it 
shculd contain just a 5 in Column 4. , 



Finish Titne:^ 
Score: 



(No. of checks 
in Yes column)' 



*Keypunch not available 
**After this record sheet has been completed, Performance Test Record Sheet f/3d/3f 
(Mount and Dismount Printer Forms) Item 8 should be completed. 
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Figure Z. Excerpt from Computer Operator Performance Test 
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ESTIMATING THE RELIABILITY OF THE TESTS 

The first empirical evidence that the tests were going to prove highly 

reliable was provided by the reliability coefficients obtained for tlie tryout 

version of the te^ts, by correlating one tryout form with the other and then 

treating these forms as half-tests and getting the reliability of the total. 
4 

These coefficients ranged from .84 to .96, with most of the tests having 
a reliability exceeding .90. However, reliability of the tryout forms, is 
obviously of less concern than reliability of the final form. To obtain the 
latter, two different approaches, were used. The first was to rescore thfe 
tryout data, to obtain scores based only on those items retained in the final 
form. The 3econd was to use the "validation study data" (the data obtained 
by administering the final forms of the tests to employees in appropriate 
jobs). 

Neither of these approaches is perfect, but each has its own advantages. 
The rescoring approach gives somewliat more stable results since the numbers 
of cases are larger. However, it has the disadvantage that because the try- 
out da£a played a part in the selection of the final items, there may be a 
slight spurious element in the reliabilities based on rescored data. 

The validation data (employee data) are entirely free of- this problem, 
but they have the major drawback that the numbers of cases for many of the 
tests are quite small, and the further disadvantage that each examinee took 
only one part of the test, thus necessitating more assumptions in estimating 
reliability coefficients for total scores (the sum of the two parts), and 
probably causing at least a slight spurious increase in some of the coeffi- 
cients. 



* For these reliability coefficients, as for all subsequent reliabilities 
obtained by correlating half-tests, the correction formula used was Angoff 
Formula 16 (Angoff, W. H. Test reliability and effective test length. 
Psychometrika , 1953 , 18 , 1-14) wherever possible rathetf than the more 
familiar Spearman-Brown formula, because the Angoff formula is somewhat 
more accurate, not requiring the usually incorrect assumption that the two 
test Italves have equal standard deviations. 



The reliability, coefficients for each test (final form) are shown in 
Table 4 along with the number of cases on which they are based and the number 
•of items in the test. 

0 

sizable discrepancies between corresponding reliability coeffi- 
cients, though partly due to sampling errors and partly artifactual in 
nature, are probably due primarily to differences in variability of the two 
groups. 



22 



TABLE A 

"Reliability Coefficients and Related Data 





Nuaiber of 
Item a 

final f< 


teat 


Data bfloed on students In field- teat 
try out (rcarored as final for*) 




Data baaed on at 


t 

iplovaea In validation i 


tudy . 






In 
arm 




Corr. 
Hot we en 


ftellab. for 

ToUl ! 
fl 4 II) 


1.0. 


No. 


of 


.Standard 


















No. of 
Student a 


of 


Employee a 


Deviation! 


Reliability Coefficients 










Tarts 
I & II 

w' 


Based 

.on 

COl • D 




Total 
<*) 














Total* 




feat Naae 


Tart 
1 

(2) 


Part 

II 

P) 


Total 

c« 


Spilt-* 

HO IE 


Part 

1 

(10) 


Part 
11 

<U) 


Part 

a I 


Part 
II 

(13) 


Part 
I 


Part Based on J 
II .Part I 

J15)° V(U) 


laaed on 
Part II 

(17) 


Agricultural Chealcala 
App lie at lone Technician 


S3 


53 


106 


m 


• 80 


.89 


.94 


16. 35 








Tmrm |>"n i ■ i -— - n t Mechanic 


54 


54 


108 


166 


.86 


.93 


.96 


•21.45 










56 


56 


112 


168 


• 88 


.94 


.95 


IS o 


52 


43 


5.74 




.77 


.65 


.87 


.79 


Word Proceaainl Specialist 


ISO 


245 






.95 


! — :w- 


78 


78 




•90* 


Apparel Galea 


4? 


49 


98 


98 * 




. .64 


.79 


'?° 


11 4.1 

11* •! 








Fabric Salea 


50 


51 


101 


98 


.73 


.85 


.93 


. A at 

14*12 








Grocery Clerk 


55 


53 


108 


1S8 


• 88 


.94 


.92 


17.00 


46 


6.03 


.78 




.87 




Hotel (Hotel) Profit Office 


S3 


53 


106 


260 


• 78 


.87 ^ 


.93 


1 1 CI 

1 / • oe 








Dental Aanlntant 


56 


56 


U2 


332 


on 

•yu 


.95 


.96 


21.91 


90 


71 


5.40) 


1 5 ;45 


.74 


.69 


.85 


.82 


Phyalcal Therapist Aeelatsnt 


54 


54 


108 


225 


* .89 


.94 


.93 


16.80 


33 


47 


5.31 


6.12 


•53 


.78 


.69 


.88 


Diatom Sewing 


50 


51 


lot* 


98 


.83 


.91 


.96 


18.20 








Restaurant Service {waiter, 
Waitress. Ceshlrr) 


61 


61 


122 


204 


.85 


1 .92 


.97 


26.38 


22 


34 


9.59 


7.93 


.87 


.90 


.93 . 


.95 


Electrunlca Technician 


S3 


S3 


106 


328 


.92 


.96 ' 


.96 


22*58 


144 


112 


8,33 


7.62 


.11 


.83 


.94 


,91 


Water Treatment Technlclsn 


45 


45 


90 


239 


.81 


.89 


.95 


16*77 


80 


75 


6.31 


6.70 


.82 


.83 


*90 


.91 


Wastewater Treataent 

Technician 


S3 


53 


106 


239 


.75 


.86 


.95 


18*37 


77 


79 


4.73 


5.13 


•M 


.80 


.91^ 


.89 


Carpenter 


55 


55 


110 


282 


.90 


.95 


.96 


21*02 


50 


13 


1 7.01 


f 


.81 


1 


.89 


t 


Diesel Mechanic 


54 


54 


108 


179 


.92 


.96, 


• 94 


20*28 


53 


29 


8.15 


6*20 


.87 


.68 


.93 


.81 


^hls U the approxiaattt reli 


ability for Part 1 an 


d Cor Part 


11. It 


la ana,lo|oua to "parallel foraa 


reliability. H ~ 













°thl. la the approximate reliability for Pert I end tor Part 11. It la aneio.oua to -parallel toraa rei».on»j. - 

•■Correlation between part, corrected by Ao.of f ' formula 116 (Angoff, W.H. Teat r.li.bility and aff.ctlv. t.at Un,th. P. T ch— trlka , 1953, W, 1-1*) 

. .. . re- £ *%t 



c Split-half reliability coefficients, corrected by Angoff foraula #16 
d rart I or Part II rcllnblllty (froa coluama 14-15) corrected by Spearasn-8rovn foraula to five reliability of the totsl tsst. 



^Spllt-hslt* rsllsblllty based on totsl test 



Too few caaea 
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TEST VALIDATION 

The extent to which the tests were valid was investigated by deter- 
mining: • 

1« How well the information and tasks measured by the tests 
matched job requirements (content validation). 

2. How well performance on the tests related to presence or 
' absence of training and '£0 job experience* 

3. How well performance on the tests related to performance 
on the job. 

In addition, analyses were performed to determine the relationship between 
test performance and course grades. , , 

Relevance of Test to Job Requirements 

.Of the major validation strategies, the matching of the test with 
industry job requirements was considered the primary thrust • For this vali- 
dation, employers across the country were contacted and invited to partici- 
pate as reviewers of the test outlines in their specialty area. A wide 
variety of sources were tapped to identify employers who would be appropriate 
content reviewers. Professional assobiations, trade associations, and numer- 
ous contacts suggested by leaders in the various fields were used. Each 
employer selected supervisors or other persons familiar with the job to 
review the performance test titles and the major areas of the job knowl- 
edge test for relevance to industry/trade requirements. Overall, the 1,600 
reviewers represented all 50 states. The rating instrument .used was a four- 
point scale that was applied to each performance test title and each area 
of the job knowledge test. The results of this content validation and the 
number of employers participating in the validation /of each test are shown 
in Table 5. It can be seen that all of the tests have been judged quite 
relevant to industry/ trade requirements, with most tests being rated 3.4 
or above where 3 equals fairly important and 4 equals very important. 

Relation of Test Scores to Training Status and Job Experience 

During the field testing phase the students taking the job knowledge 
test included some examinees who had been in a training program for the 
vocational area covered by the test and some who had had no such training. 
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TABLE 5 

Content Validity of Job Knowledge and Performance. Tests 



Test 
No. 



Vocational Curriculum 
Areas and Test Names 



Mean Rating a 



Job Knowledge 



Performance 



Number 
of 

Reviewers 



Number 

of 
States 



Agriculture 



11 
12 

21 
22 

31 

33 
34 

41 
42 



51 
52 



61 
62 
63 



71 
72 



Agricultural Chemicals 
Applications Technician 

Farm Equipment Mechanic 
Business and Off 'fee 



Computer Operator 

Word Processing- Specialist 

Distributive Education 



Apparel 'Sales 
Fabric Sales 



Grocery Clerk 

Hotel (Motel) Front Office 

Health 



Dental Assistant ' 

Physical Therapist 
Assistant 

Home Econom ics 

Custom Sewing 

Restaurant Service 
(Waiter, Waitress, Cashier) 

Technical 

Electronics Technician 

Water Treatment Technician 

Wastewater Treatment 
Technician 

Trade and Industry 
Carpenter 
Diesel Mechanic 



3.4 
3.3 

3.0 
3.7 

3.6 
3.4 
3.5 
3.7 

3.6 
3.5 

3.6 
3.5 

3.5 
3.3 
3.4 



3.3 
3.6 



3.4 
3.4 

3.0 
3.4 

3.5 

2L2_ 



3.3 
3.2 

3.5 
3.7 

3.6 
3.5 



2.9 

3.4. 

3.4 



3.5 
3.5 



77 
74 

61 
110' 

46 

-28— 
173 
73 

158 
59 

21 
83 



193 
91 
130 



81 
143 



"26- 
20 

19 

33 

12 

-1-3- 
34, 
28 

38 
12 

10 

27 

36 
21 
24 



23 
39 



4 
3 
2 

ERJC 



Very important 
Fairly important 
Of Minor importance 
Of No importance 
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It was hypothesized that if the tests were valid the trained group should 
score higher on the job knowledge test than the untrained, and this in fact 
proved to be the case. The first part of Table 6 summarizes these data. 

Varying amounts of job experience were represented in the employee 
groups tested. It was hypothesized that if the tests were valid the groups 

-wit h-^aab s_ta n 1 1 a 1 _am ounts of experience would score higher than those with 

* ~~ ~ "~ — ~ — — - . 

less experience. For most of the tests wHer^d~ata-were-available ^ this 
proved to be the case, providing another kind of evidence that the tests are 
valid. Table 6 also summarizes the results of these analyses. 

r 

Not only does Table 6 provide evidence of the tests' relationships to 
training status and to job experience level, but when the "trends" across 
the— trabte"for~e"ach test are examined, the increase in mean test score is in 
the expected direction; that. is, one would expect non-trained students to 
have scored lowest and more experienced workers to have scored highest. 
This is, in fact, what generally occurred for' the majority'of the tests. 
Even the deviations from this expected trend are slight and the overall pic- 
ture from Table 6 is a further indication of test validity. 

Relation to Job Performance 

The previous section discussed the relationship between' test 
performance and the objective criteria of training status and relevant job 
experience. In addition to these data, subjective evaluations of job per- 
formance were obtained from supervisor's. Since the tests had been designed 
as modular, employers selected various performance tests to administer to * 
some of their employees. For the most part, the selected employees took the 
entire job knowledge test and several performance tests. Supervisors were 
asked to rank these employees in terms of the quality of their job perfor- 
mance, and these rankings were used as the measure of job performance. 
Because of the burden imposed by extensive testing in the job setting, par- 
ticipation in the employee testing validation was less than initially hoped 
for* , Accordingly, for six of the tests it was not possible to relate test 
performance with supervisors 1 rankings. It is interesting, however, tp 
examine the findings for those tests for which we were able to obtain 
'sufficient employee data. Table 7 shows the^correlations between total jot) 
knowledge test score and job performance rankings. 
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TABLE 6 



Mean Job Knowledge Test Score, Student Training Status, 
and Employee Experience Level 



1 




Student Training Scatus 




Employee 


Job-Related Experience 




I 

Test 


Vocational Curriculum 


Non-Trained 


Trained 


Under 1 yr . 


1 


-5 yrs. 


Ov.er 5 yrs. 


Unknown 


No. 


Areas and Test Names 


N 


Mean 


N 


Mean 


N 


Mean 1 


«j 


Mean 


N 


' Mean 


N 






Agriculture 
























11 


Agricultural Chemicals 
Applications Technician 


50 


57.0 


130 


65.3 










•> 
J 


00. J 




12 


Farm Equipment Mechanic 
Business and Office 


44 


41.5 


141 


59.2 














- 


21 


Computer Operator 


. 36 


43.7" 


152 m 


70.6 


10 


82.9 


37 


82.5 


16 


. 87.9 


29 


22 


Word Processing Specialist 
Distributive Education 


49 


87.1 


204 


105.6 


27 


112.1 


36 


119.0 


8 


108.0 


7 


31 


AppareL Sales 


24 


* 

55.4 


100 


58.7 


* 




4 


62.0 


ix 






32 


"Fabric Sales 


'24 


57.8 


100 


66. 1 










a 




* 


33 


Grocery Clerk 


36 


57.5 


126 


61.0 


6 


76.8 


18 


78.2 


22 


85.0 


O 


34 


Hotel (Motel) Front Off ice 
Health 


44 


48.0 


212 


60.0 


M 








— 


** 


* 


41 


Dental Assistant 


59 


41.3 


275 


77.6 


15 


76.9 


55 


78.8 


27 


84.8 


31 


42 


Physical Therapist 
Assistant 

Hoiao Economics 


41 


47.3 


185 


76.8 






27 


73.5 


20 


67.9 

- 


20 


51 


Cuscom Sewing 


24 


50.4 


100 


64.5 






— 










52 

- 


Restaurant Service 
(Waiter, Waitress, Cashier) 

Technical 


41 


69. 3 


181 


76.2 


A 


88.5 


13 


91.2 


22 


91.6 


3 


61 


Electronics Technician 


51 


37.0 


296 


62.1 


3C 


63.4 


79 


66.9 


56 


75.8 


43 


62 


Water Treatment Technician 


27 


33.8 


252 


53.3 


9 


58.1 


29 


61.1 


25 


64.4 


12 


63 


Wastewater Treatment 
Technician 

Trade and Industry 


27 


43.1 


252 


70.2 


3 


82.0 


67 


80.0 


32 


80.9 


33 


71 


Carpenter 


53 


54.7 


221 


70.6 


5 


66.6 


16 


77.0 


12 


73.9 


27 


72 


Diesel Mechanic 


33 


51.2 


145 


78.0 


5 


73.0 


28 


76.4 


20 


83,6 


29 



a Too few cases 
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TABLE 7 

* / 
Correlation Between Total Job Knowledge Score and Job Performance Ranking 



Test 
No* 


Vocational Curriculum 
Areas and Test Names 


Correlation 


Number of 
Employees 


Number of ' 
Sites 




Agriculture 








11 


Agricultural Chemicals 
Applications Technician* 


a 






12 


Farm Equipment Mechanic 
Business and Office 


a 






21 


Computer Operator ' 


.35* 


76 


15 


22 


Word Processing Specialist 
Distributive Education 


.48** 


65 


12 


31 


Apparel Sales 


a 






32 


Fabric Sales 


a 






33 


Grocery Clerk 


.19 


46 


8 


34 


Hotel (Motel) Front Office 
Health 


a 






41 


Dental Assistant 


.24* 


114 


23 


42 


Physical Therapist 
Assistant 

Home Economics 


.04 


58 

• 


12 


51 


Cus t om S ewinc 


a 






52 


Restaurant Service 
(Waiter, Waitress, Cashier) 

Topnn A r* o 1 
L cLIUllLdl 


.33 


41 


7 


61 


Electronics Technician 


.43** 


182 


28 


62 


Water Treatment Technician 


.76** 


73 


15 


63 


wastewater Treatment 
Technician 

Trade and Industry 




131 


21 


71 


Carpenter 


.22 


39 


6 


'72 


Diesel Mechanic 


.25 


59 


11 
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*Significant at ,05 level 
^Significant at .01 level 

a Number of cases too small to compute correlation 

NOTE: The value of the correlation coefficient shown in this table is the weighted 
average of the separate within-site correlations (weighted by the number of* 
cases). " 
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Altogether, a tdtal of 884 individuals participated in this validation 
effort. Included here were 348 Air Force uniformed personnel from 50 bases 
and 26 Navy uniformed personnel from 7^sites. The number of employees par- 
ticipating, by test, is shown in "the table along with the number of sites 
involved. Of the 11 coefficients computed, 6 were significant at the .05 
level or beyond. 

Various considerations in regard to the rankings "made it undesirable to 
treat numerically equal rankings from all "sites as equivalent. In the first 
place is the fact .that the top-ranking performer at one site might be among 
the lowest-ranking at another. Furthermore, a ranking of, say, "5". at a 
site where 40 employees were being ranked would have quite a different mean- 
ing from the same ranking where anly-'five were being ranked* To avoid these 
problems, correlation coefficients were obtained separately within .each site 
and in effect "averaged" across sites. 

A word of caution is in order when interpreting these correlations. 
There are well-recognized limitations o£ subjective ratings and rankings as 
criteria, and the correlations of the performance assessment with the test 
scores could, in some cases, be better regarded as evidence of whether the 
supervisors 1 assessments themselves possess any validity. 

Correlations with Course Grades 

During the field test, course grades in relevant courses were provided 
for the examinees by some of the schools. As shown in Table 8, the correla- 
tions between these grades and total- scores on the tryout forms were substan- 
tial, ranging between .34 and .68. 



This is equivalent to computing a single overall correlation coefficient 
in which the values correlated are not raw rankings but rather standardized 
variables with a mean of 0 and a standard deviation of 1 for each site. 



o i 
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TABLE 3 . 

Correlation Between Course Grades and Total Jo,b Knowledge Score 



a 

Test 
No. 



1 

2 

3 
4 

5 
' 6 



7 
8 



9 
10 



11 
12 



13 
14 



Vocational Curriculum 
Areas and Test Names 



Corre- 
lation 



Agriculture 

Agricultural Chemicals 
Applications Technician 

Harm .Equipment Mechanic 

Business and Office 

Computer Operator / 

Word Processing Specialist 

Distributive Education 

Grocery. Clerk . 

Hotel (Motel) Front 
Office , 

Health 

Dental Assistant 

Physical Therapist 
Assistant 

Home Economics 

a 

Fashion/Fabric Sales 
and Sewing 

Restaurant Service (Waiter 
Waitress, Cashier) 

Technical 

Electronics Technician 

Water/Wastewater a 
Technician 

Trade and Industry . 

Carpenter 

Diesel Mechanic 



.34 
.59 

• 46- 
.42 

.51 
.45 



.62 
.46 



.68 
.58' 



.43 
.63 



.42 
.40 



Number of 
Course 
Grades 



81 
172 

53 
95 

43 
140 



120 
352 



30 
49 



352 
61 



114 
97 



Number of 
Separate Corre- 
lation Coefficients 
Entering into the 
Overall Correlation 



3. 
7 

2 
8 



10 

27 



3 

5 ' 



20 
5 



6 
6 



9 
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a Field test tryout form. Tests were numbered 1 thivough "14. The items in 
Test 9 have now been split into £hree tests: Apparel Sales, Fabric Sales, 
and Custom Sewing; while the items in Test 12 have been split into 
Water Treatment Technician and Wastewater Treatment Technician. 

b Correlations were computed separately £or each site and course, and then 

averagfed, weighting each correlation by the corresponding value of N-3 
' where N equals the number of examinees 1 course grades entering into the • 
correlation. ' 
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WORK HABITS INVENTORY 



Development 

The Work Habits Inventory was designed as a means for raising the com- 
petence level of students in job survival skills in the areais of positive 
work values, habits, and attitudes* Such skills are often not stressed in 
the more technical aspects of the vocational program curriculum. The use of 
the instrument should improve the communication between the student and the 
teacher wi.th respect to. the important "nontechnical" skills and can provide 
the basis for instruction and cpunseling t in these areas. 

* ** 

f 

In order to identify possible work habits to include in the inventory, 
an intensive review of the literature was undertaken, beginning with a com- 
puter search. Following. the literature review, project staff undertook the 
development of (1) format, (2) general behavioral dimensions, and (3). speci- 
fic items for the Inventory. Items were developed to tap the following gen- 
eral areas: 

- being dependable 

- giving an honest day's work 

- knowing what is expected of you 

- maintaining good health 

- managing time and materials efficiently 

- getting along with people with a variety of 'personalities 

- working as a team member, when appropriate 

- knowing your own abilities, strengths, and weaknesses 

- being loyal to the organization for which you work 

- making independent decisions, when appropriate 

- using initiative and imagination 

- working without close supervision 

- working under tension or pressure 

- adjusting to various work situations 

- being honest t 

- persevering 

- having appropriate personal appearance — 



The draft versions of the Inventory .were reviewed by several members of 
the project's Subject Matter Review Panel and were pilot tested. Following 
the pilot test, items and response format were modified as necessary. The 
revised Inventory was then field tested in the schools participating in the 
administration of the "technical" portions of the tests. 




Validating the Inventory was done by determining the relevance of the 
items of the Work Habits Inventory for each job by asking the same* employers 
who completed # the rating forms for the .content validation of the job knowl^ 
edge and performance tests to also participate in validating this Invehtory. 
Because it was thought that the various statements on' the Inventory would 
have different relevance for different jobs, analyses were performed to 
determine the importance of each separate item for each occupation measured 
by the tests* Again, a four-point scale was-used by each reviewer. 
(4 » Very Important; 3 ■ Fairly Important; 2 » Of Minor Importance; 1 » Of 

No Importance). The results are shown in Appendix C, along with a copy of 

* / 

one part of m the final Work Habits Inventory identifying the individual items. 
While an examination of this table will reveal that the occupations have * 
difffer&nt sets. of "Most Important" items, the desire was to have one instru-, 
ment chat* crossed over all the occupations tapped by the job knowledge and 
performance tests with a separate key of important work habits provided- as 
part/of the test package for each occupation. 



l^/u 



There are three parts to the Inventory: one is completed by the student 
abbut his or her own traits, titled "How I Am"; another part, entitled ' • 
"^Importance of Job-Related Traits," also completed by the student, asks the 

^student to estimate the importance of the job traits from an employer's point 

/ 

of view; and a third part, "Assessment of Student's Work Habits 1 ," is com- . 
pleted by the teacher about each student. All three parts contain essen- * 
tially identical items; the viewpoints from which they are answeredare 
different as indicated above. The three parts of the Work Habits Inventory 
are contained in the, test package. 

Use of Inventory 

It is recommended thaca three-step procedure be used. These steps are: 

1. Administer the "Importance -of Job-Relatett Traits" Inventory 
to new students in a vocational 1 program. The students 
should indicate how important "they ttrink each behavior will 
be to their future employers. The Inventory would be 
"scored" using the appropriate "job key." 

* 2. The results of this administration can then serve as the 

basis for instruction on proper pork habits. This instruc- . 
. tion should ensure 'that students know what the employer in 

the specific job. -area expects. 



3. Later on in the school year, the student should rate his or 
her ovm behavior on the items in the "How I Am" Inventory 
and, at about the same time, the instructor would do the 
same for each student, using the "Assessment of Student 1 s 
Work Habits" part o c f the Inventory* These individual stu- 
dent self-ratings can then be compared with the instructor 
ratings and the appropriate "scoring key,"*and together 
they would serve as a communication tool between student 
and teacher or between student and counselor* 



4: 
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PROJECT DISSEMINATION 



The third objective of the Vocational Competency Measures project — to 
promote the acceptance and use of the occupational competency tests — required 
a nationwide dissemination/diffusion effort which is the subject of this 
section of the report* In this section, we describe the specific activities 
carried out to achieve the dissemination objective* 

Underlying our effort was a basic philosophy that dissemination is a 
critical activity that continues throughout the pro ject—fcpm beginning to 
end and hopefully beyond,* It is an ongoing effort that is automatically a 
part of other tasks. Every individual we contacted, every meeting we held 
or attended (whether formal or informal) to discuss the project was a form 
of dissemination* Contacts made as a part of other tasks, whether they were 
for occupational selection, verification of competency requirements, or field 
testing, were also part of the dissemination task. For each new contact, we 
provided a description of the project. In turn, these key individuals often 
served as referral links to leaders in other organizations. These kinds of 
activities occurred throughout the course of the project as a part of every 
major task* 

Also critical to our dissemination effort was an overall strategy of 
involving users early and keeping -the field informed * Individuals involved 
in the development 6f a project, who are kept informed about activities 
throughout the project, .develop a personal interest and feel a part of it, 
thereby increasing the likelihood that the products of the project will be 
accepted and used* Throughout the project, many organizations requested 
further information about the competency tests* Some became actual partici- 
pants in the field t£st of our measures. All were placed on our distribution 
list to receive periodic information bulletins on major project milestones. 
This list eventually grew to over 3,000 names* 

The dissemination activities for the project involved a combination of 
people and materials* These activities are listed below and described in 
the paragraphs that follow: 
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• Project Abstr^t 

• Project Information Brochure 

• State-of-the-Art Report 
' • Project Update 

& Journal Articles and Newsletters 

t American Vocational Association Conventions 

• Targeted Presentations at Meetings of Educators 

• Technical Assistance to the States 

r 

t 

Project Abstract 

The project abstract was one of the very first dissemination activities 
to be completed by the project. It was submitted to the ED Contracting 
Officer during the first: week of the project and was also" made available to 
those requesting project information until the project information brochure 
was ready for dissemination. In addition to brief, identifying information 
on the project and the contractor, the abstract included an outline of pro- 
ject objectives, procedures, and expected contributions to education. 

Project Information Brochure 

The project information brochure was also one of the early dissemination 
activities" completed by the project. It was designed for use by project 
staff and ED in responding to inquiries and in creating an awareness of 
project objectives, activities, and outcomes. The brochure was a highly 
appealing and significant dissemination vehicle and it served that purpose 
extremely well. We received many compliments on it. 

The brochure provided a major contribution to project visibility and 
significantly enhanced our dissemination efforts. Over 15,000 copies were 
distributed. In addition to serving as a handout for interested individuals 
and as an enclosure in information mailings, it facilitated the ease with 
which various project tasks were completed. For example, students who par- 
ticipated in the pilot testing of the Work Habits Inventory received a bro- 
chure and proudly showed it to their classmates as a sign of their involve- 
ment in an important educational research project. Also, job incumbents and 
supervisors who were considering participating in test validation appreciated 
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the. importance of the project and their critical role in it as a result of 
reading the brochure. 

State-of-the-Art Report* * 

* « 

One of the major tasks that began during the early months of the project 
was a review of resources in the field pf occupational competency measure- 
ment. While the major purpose of this review of previous, research and 
development was to ensure that project staff capitalized on the latest 
Experiences in developing and evaluating occupational competency measures, 
the highlights of this search were-docuraented to assist others who are work-? 
tng, <?r are planning to work, in the field of competency measures for voca- 
tional education. Specifically, the intent was to provide a review of occu- 
pational competency testing, including a summary of the AIR project as well 
as other major efforts under way today and some of the methodological devel- 
opments that should be of interest to those working in this area. Technical 
and legal considerations in setting test score standards are also discussed 
and an extensive list of references is included. This report has been dis- 
tributed widely throughout the country, with over 2,000 copies disseminated 
by December 1982. 

Project Update y 

Our project "Update" was a single-sheet information bulletin on impor- 
tant project milestones. It was a useful tool for keeping potential users 
Informed of current activities. Fourteen issues were prepared and distrib- 
uted periodically to all ttiose on our distribution list. As mentioned pre- 
viously, over 3,000 names appeared on this list. In addition, we received a 
number of indications that the "Update" was reproduced and copies forwarded 
to other individuals. 

Journal Articles and Newsletters 

Journal articles were submitted throughout the project both in response 
to specific requests from editors and on a staff-initiated basis. The pur- 
pose of the articles was to communicate project information to potentially 

0 Chalupsky, A. B. , Phillips- Jones, L#,,& Danoff, if. N. Competency mea- 
surement in vocational education: A review of the state of the art . Palo 
Alto, Calif: American Institutes for Research, June 1981. (A1R-81914- 
6/81-RPI). (ED 205 715) 
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interested individuals who might not be reached by other dissemination 
methods. Articles about the project appeared in seven journals. 

Ano the ^effective means of keeping the field informed of project 
activities was through news releases to various newsletters. Several bene- 
fits were derived from having project information publicized in the communi- 
cation cha me Is of interested groups: (1) these large-audience publications 
extended the contacts beyond those the project already had, (2) the pool of 
potential test sites was expanded, and (3) increased project visibility 
was gained which will enhance utilization of the competency tests. 

A project brochure and a news release summarizing current project 
activities were sent to editors of appropriate educational* and professional 
association newsletters. Articles about our project appeared in at least 25 
publications that we know of. Appendix D presents a listing of journals and 
newsletters that have carried articles on the project. 

American Vocational 'Association Conventions 

The American Vocational Asosciation conventions attract- a large number 
of people involved in vocational education, including those specifically 
interested in occupational competency testing. The conventions provided an 
excellent opportunity for us to distribute materials about the project, give 
formal presentations, hold informal gatherings, and meet individually with 
key people. 

In December 1979, the Project Director presented a description of the 
project to the American Vocational Association Convention in Anaheim, 
California. The invited presentation was part of a theme session entitled 
"Are Our Students Ready for Work?. • .Measuring Competency." Other dissemina- 
tion activities at the 1979 convention included distribution of project 
brochures at key exhibit booths and numerous personal contacts with individ- 
uals committed to occupational competency testing. The first meeting of the 
4 project 1 s National Policy Council was held immediately following the 
convention. 



■ In December 1981, project staff participated again in the AVA Convention 
in Atlanta, Georgia. The Project Director made two presentations. He par- 
ticipated in 'the New and Related Services Division Carousel of Special 
Interests, and, also presented a status report of the project the Vocational 
Instructional Materials Section of the New and Related Services Division. 

Targeted Presentations at Meetings of Educators 

Our original dissemination plan proposed a three-day national informa- 
tion workshop to be conducted in the third and final year of the project (as 
specified in the Request for Proposal), with the expenses of approximately 
163 workshop attendees paid for with project funds. After careful analysis, 
staff felt that the extremely wide diversity in the- subject matter of the 
project tests and the fact that some of the tests are applicable at high 
school level, others a't postsecondary or adult levels, and still others 
across the entire grade range required that the dissemination strategies be 
modified to be fully responsive to this diversity. 

Accordingly, AIR proposed (and it was approved by ED) that, rather than 
have one large dissemination workshop, project staff make targeted presenta- 
tions at meetings of educators who are directly concerned with one or more 
of the fields encompassed by our competency tests. To the maximum extent 
possible, these presentations were made at meetings already scheduled as 
part of professional association or technical specialist gatherings. 

A prime focus of our project dissemination efforts was national or 
regional meetings of state leaders in various aspects of vocational educa- 
tion. Examples of such meetings where project staff made briefings include: 

• New York State Commissioners Conference for Occupational 
Evaluation Directors, January 1980 

• National Network for Curriculum Coordination in Vocational 
and Technical Education (NNCCVTE), July 1981 and July 1982 

• Michigan School Testing Conference , .March 1982 

• The Utah Vocational Education Conference at Utah State 
University,; March 1982 

• Regional Coordinating Unit (RCU) Directors Annual Meeting, 
April 1982 
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• Summer Professional Improvement Conference for Vocational 
Education in Massachusetts, June 1982 

• Nineteenth Annual Pennsylvania Department of Education 
Curriculum and Instruction Conference, July 1982 

• New York City School District— Technical Assistance Program 
for Senior Vocational Education Administrators, July 1982 

• Twelfth Annual All-Service Vocational Education Conference, 
North Dakota, August 1982 

• Statewide Vocational Education Conference, Nebraska, August 
1982 

• Annual West Virginia Vocational Teachers Conference, August 
1982 

• Texas State Technical and Industrial Teachers 1 Workshop, 
August 1982 

• Annual Vocational Education Workshop, Florida, August 1982 

• National Joint Apprenticeship and Training Committee for 
Operating Engineers, September 1982 

• East Central Network for Curriculum Coordination, September 
1982 ■* 

• Training, Education and Research Subcommittee of the 
Military Interservice Review Organization, September 1982 

• Louisiana State Department of Education Staff meeting, 
September 1982 

• Idaho Vocational Educators Conference, (Distributive 
Education and Business and Office), October 1982 

• Arizona State Department of Education Staff, November 1982 

Briefings on the project were also presented to the U.S. Department of 
Education and other interested groups in Washington, D.C. in December of 
1981 and August of 1982. 



In addition, project staff made presentations at some 30 professional 
association meetings, as shown in Appendix D. Wherever appropriate, we sup- 
plemented these presentations by meeting with cognizant staff members of 
state education agencies to provide technical assistance in thef area of stu* 
dent competency testing. 



x With this revised strategy, we achieved far more effective coverage— 
both geographically and technically—than would have been possible under the 
originally scheduled single meeting. In fact, we were able to participate 
in meetings across the country, covering all regions. We accomplished this 
strategy at virtually the same cost that was specified in the original work 
statement ♦ ^ 

Technical Assistance to the States 

Technical assistance was provided to state and local education agencies 
largely in conjunction with briefings as part of our overall dissemination ' 
meetings as noted above. A good share of our dissemination presentations 
were to states recognizing the need for further training in the area of com- 
petency testing and which requested our participation in their staff 
meetings. 

No amount of "pressure" on AIR's part could substitute for this willing- 
ness of a sufficient number of educators in a state to begin preparations 
for such testing* Accordingly, we avoided an arbitrary quote on in-depth 
technical assistance to one state in each ED region, as was originally 
planned. Instead, we made our project and its products known to every state 
and then provided whatever support services we could within the contract 
trime and budget, upon state request. 

One set of products resulting from the project will be particularly 
valuable both from the standpoint of technical assistance and from our goal 
of fostering test development on a self-supporting basis upon completion of 
the project. These were the four manuals designed to help vocational educa- 
tors not only in improving test usage but also in developing and field test- 
ing new measures. At the request of the director of the East Central Cur- 
riculum Coordination Network, it was decided to adapt these manuals to the 
VECS (Vocational Education Curriculum Specialist) format for publishing by 
this Network as part of the VECS series developed under a previous contract 
with the U.S. Department of Education. As a result, the dissemination intact 
of the project will continue long after the current contract ends. A 
description of the manuals is provided in the next section. 
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PLANNING FOR CONTINUING COMPETENCY TEST 
DEVELOPMENT AND ADMINISTRATION 

Field Test Site Feedback 

During the field test scheduling and data collection stages, AIR pro- 
ject staff maintained continuing telephone and mall contact with the desig- 
nated representatives of field test schools. In addition to assisting school 
representatives solve both the technical and logistics problems associated 
with test administration and helping In the revision of the individual 
tests, the* field test feedback — looked at across all the tests — provided 
some valuable general Information that should be taken Into account in any 
future test development efforts. Among the "lessons" learned for achieving 
cooperation of field test participants were the following: 

• Teachers are extremely busy and no amount of hlgh v level, 
administrative approvals will ensure that testing Is accom- 
plished If the teachers are not truly committed to vocational 
competency testing and the resulting educational benefits 
for their students. 

• Teachers who are really committed to the measurement of 
vocational competency will find the time even if (or espe- 
cially If) they f re not pushed by the school or district 
administration. 

• Competency testing must be kept to a reasonable "time period 
without excessive encroachment on teaching time. ■ Ideally, 
the time speat in testing will be viewed by teachers as a 
vj.tal part of the educational process. 

• Test content should be obviously relevant to both teachers 
and students. 

• Test administration instructions must be kept as simple as 
possible* 

• For teachers interested and willing to test, but who really 
do not have time during the regular school day, arrangements 
for payment for afterschool time (weekends, etc.) should be 
made. 

• Procedures for handling the tests before, during, and after 
the testing should be as simple and straightforward as 
possible. 



Suggestions for New Test Development 

From the early days of the project through the- field test and 

validation stages, we have continued to receive suggestions as to 

6ther tests that are in need of development* The suggestions range 

across the entire job spectrum and include the following areas: 

Drafting Secretarial Competency 

Merchandising Machine Shop Operation 

Graphic Arts Repair Parts Personnel 

Jewelry Sales Truck Driving 

Jewelry Manufacturing Other Health Related Areas 

Fashion Design including Occupational 

Interior Design , f Therapy Assistant 

These suggestions typically reflected needs on the part of school or 
employer representatives rather Uhan commitments to participate in the 
development and/or funding of new test development. In fact, based on 
informal contacts with educators, it is doubtful whether the currently 
strained state education budgets will permit any resources to be channeled 
to competency test development, at least in the near future* Even the pur- 
chase of existing tests, regardless of their quality, will very likely be at 
a level well below expectations of a few years ago» While more information 
•needs to be obtained, it appears that employers or employer associations may 
be a much better source for continuing test development than school agencies, 
at least in the immediate future* 

Follow-up Contacts with Individuals on the Project General 
Mailing List 

In order to provide prospective test publishers with some indication of 
the potential market for each of the vocational competency measures, in mid- 
December 1982, a letter was sent to approximately 3,000 individuals and 
organizations on the project's mailing list describing the content of the 
test packages, providing a tentative estimate of test costs, and asking for 
an estimate of the number of test packages they think they will need in the 
near future. In this same mailing, individuals were asked whether they 
would be interested in cooperating in future test development efforts. 

Within two weeks after mailing this letter, approximately 40 
responses had been received despite, the fact that the letters 
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arrived near the peak of the Christmas holiday preparations. Of those who 
had responded by late December, eight (20%) indicated interest in learning 
how their organizations can help in the development "and/or funding of new 
tests. We expect that by the end of the VCM project we will have received a 
sizable number of responses indicating interest in participation in future 
test development as well as in the use -of the current AIR tests* 

Forwarding of Specimen Test Packages to State Departments of 
Vocational Education 

In keeping with the high priority placed on dissemination in this pro-? 
ject, it has been decided to forward a complimentary set of all 17 test 
packages to each State Department of yocational Education (including the 
District of Columbia and the Education Departments in the outlying territo- 
ries). This will be done before a publishing agreement has been completed 
with a test publisher. 

The advance dissemination of the test packages will enable State Direc- 
tors and their key staff to examine the tests and come up with sound esti- 
mates, of how many tests they will need when the tests are published,, These 
estimates will then be combined with information received from Individuals 
on our general mailing list and vill be made available to prospective pub- 
lishers of our tests. 

The views of State Directors of Vocational Education concerning their 
needs for new tests and their interest in being involved in future test 
efforts should also be helpful for future competency test development. 

Assisting Other Agencies in Developing Vocational Competency Measures 

During the early stages of the project, it was envisioned that project 
dissemination would not only provide technical assistance on using competency 
measures, but also on the development of such measures. As it turned out 
nearly all the education agencies requesting project assistance were much 
more interested in becoming aware of what was available or soon to be avail- 
able in the field of competency measurement rather than in learning the spe- 
cific techniques of test development. Neither the interest level nor the 
resources available (personnel or monetary) indicated that such technique- 
oriented workshops would be appropriate during the period of the VCM project. 

• « 51 



Accordingly^ it was decided to expand the test development guidebook that 
had been planned since the start of the project • Rather than produce a., 
single guidebook, the decision was made to prepare four separate manuals*as 
follows: 

• Using Competency Measures Effectively 

In this module would be covered the history of compe- 

« 

tency testing, current major efforts now underway in the 
field, important definitions, and the many appiications of 
vocational competency measurement in vocational education. 

• Determining Requirements for Vocational Competency Measures 

Covered in this module would be answers to the question 
"How do you determine what to measure?" including step-by- 
step procedures for collecting job and task, information and 
what should be done with all the information. 

• Developing Vocational Competency Measures 

Included in this module would be a discussion of the 
important considerations in test development and "an outline 
of the procedures for designing and constructing tests, 
beginning with the preparation of initial test specifica- 
tions, through item development, pilot testing, field test- 
ing, and test revision. 

• Validating Competency Tests and Using Test Results 

A discussion of the various aspects of test validity 
and an outline of procedures for determining and maintain- 
° ing test validity would be contained in this module, along 
with suggestions for reporting test results and setting test 
standards. 

It was further decided to prepare these manuals so they could be used 
not only as operational handbooks, but also as training modules. These mod- 
ules are scheduled to be published in the near future by the East Central 



Network for Curriculum Coordination 7 as part of the Vocational Education 
Curriculum Specialist series developed by AIR under a previous ED/OVAE 
project* 

i 

This series will provide "stand alone"' support to educational agencies 
and other institutions interested in learning more not only about the poten- 
tial of vocational competency measures and how to use t&em properly, but 
also how to go about- planning and developing/these measures** The manuals 
can also serve as texts or supplementary materials in training programs. 

Through the distribution network of the East Central Network for Cur- 
riculum Coordination and its linkages to the National Network for Curriculum 
Coordination in Vocational-Technical Education (NNCCVTE), the project expe- 
riences in test development will be available to all states long after the 
completion date of the VCM project. 1 • 

.» • 

Hopefully in the next few years, the technical and financial, resources 
of educational agencies will be much more capable of supporting individual 
or cooperative efforts in vocational competency test development. As of 
now, however, we know of wide-scale development and evaluation of vocational 

competency tests only in Florida- and the several states participating in the 

* 8 
Student Occupational Competency Achievement Testing (SOCAT) Consortium. 



7 For ordering and price information, contact Ms. Rebecca Douglass, . 
Director, East Central Curriculum Coordination Center, Sangamon 
State University, E-22, Springfield, IL 62708. 

^ For further information qn the test development in Florida, contact 
Dr. Roy Giehls, Program Director, Evaluation Section, Division of 
Vocational Education, Florida Department of Education, Tallahassee, 

* FL 32301. For information on SOCAT, contact Dr. Gordon McMahon, 

National Occupational Competency Testing Institute, 45 Colvin Avenue, 
Albany, NY 12206. ' ' 
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APPENDIX A 



'Members of the National Policy Council and the 
National Subject Mafcter Panel 
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National Policy Council Members 



Mr. Ingo Antonitsch 

Executive Director of City and 

County of Denver Commission of 

the Disabled 



Dr. Thomas M. Bogetich 
Executive Director 
California Advisory Council on 
Vocational Education 



Dr. Ralph C. Bohn 

Dean of Continuing*- Education 

San Jose State University 



Dr. Addison S. Hobbs, Director 
Vocational- Technical Education, 
State of Maryland 

Dr. Marion B. W. Holmes 
Director of Vocational Education 
School DistrAt of Philadelphia 

Mr. George Kosbab 

Assistant Director 

Curriculum and Staff Development 

Ohio State Department of Education 



Dr. Ralph Bregman 
The National Advisory Council on 
Vocational Education 



Dr. Donald M. Clark 
President, National Association 
For Indus try-Educatiott Cooperation 



Mr. Eustaquio Cortez 
Automotive Department 
Evergreen Community College 
San Jose, California 



Dr. Esther E., Diamond 
Senior Project Director 
Science Research Associates 
Chicago, Illinois 



Ms. Wilma Ludwig 

State Director of Vocational Education, 
New Mexico 



Ms. Judith McKeever 

Nursing Assistant Program 

916 Area Vocational-Technical Institute 

White Bear Lake, Minnesota 



Mr. Philip W. Osborne, General 

Manager (Retired), Industrial Relations 

Aluminum Company of America 

Dr. Gordon I. Swanson 

Professor of Vocational Education 

Past President, America^ Vocational Assoc. 

University of Minnesota 



Dr. Carol Eliason 
American Association of Community 
and Junior Colleges 



Mr. Reese Hammond 
Director of Education & Training 
International Union of Operating 
Engineers, AFL/CIQ 



Dr. Robert J. Thompson 

Vice President, American Vocational Assoc. 
<■ Reg ion V 

Foothill-DeAnza Community College District 
Los Altos Jiills, California 



National Subject Matter Panel 



Agriculture Education 

Mr* Paul Day, State Supervisor, 
Agriculture Education 
1 esota Staje Department of 
education 

Mr* John Murray 

Jackson Area Vocational-Technical 

Institute 
Minnesota 



Business and Office Education 
Dr* Robert Poland 

Coordinator of Vocational, Technical, 

and Applied Arts Education 
Michigan State University 

Mr* John Lee 

Chief State Consultant, Business 

Education 
Indiana State Department of Public 

Instruction 



Distributive/Cooperative Education 

Dr* Robert L. Bennett 
Director of Planning and Development 
San Mateo County Community College 
District, San Mateo, California 



Home Economics Education 

Dr'. Mary Ann Parthum 
Denver Public Schools 

Dr. Hazel Crain 

Center for Business and Vocational 

Teacher Education 
University of Nebraska-Lincoln 



Technical Education 
Dr* Robert Keck 

State Supervisor of Technical Education 
Oklahoma State Regents for Higher 
Education 

Dr. George Mehallis 

Executive Director for Technical Education 
Broward Community College 
Miami, Florida 



Trade and Industry Education 
Mr. Ross Alloway 

Ross ton Schools of Men's Hair Design 
Long Beach, California 

Mr. Robert' Patterson, Director 
Vocational Industrial Education 
Texas Education Agency 



Jls. Elinor Burgess 

County Supervisor, Distributive 

Education 
Fairfax County Public Schools, 

Virginia 

Health Occupational Education 

, * Dr. Mildred Pittman, Coordinator 

Health Occupations Education 
School of Education, University of 
Indiana 

Ms. Roberta Firetag 
San Jose City College 
San Jose, California 
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APPENDIX B 
Schools Participating in Pilot Testing 



Schools Participating In Pilot Testing 



Test Name 



School and Location 



Agricultural Chemicals 
Applications Technician 

Farm Equipment Mechanic 

Computer Operator 



Word Processing Specialist 



Grocery Clerk 



Hotel (Motel) Front Office 



Dental Assistant 



Physical Therapist 
Assistant 

Apparel Sales, Fabric Sales, 
Custom Sewing 

Restaurant Service (Waiter, 
Waitress, Cashier) 

Electronics Technician 



Water Treatment Technician, 
Wastewater Treatment Technician 



Delta Community College, Stockton, California 



Delta Community College, Stockton, California 

Richmond Regional Occupational Center, 

California 
Computer Technology School, Pittsburgh, 

Pennsylvania 

ICM School of Business, Pittsburgh, 

Pennsylvania 
Foothill College, Los Altos Hills, California 

Eden Area Regional Occupational Center, 
. San Lorenzo, California 

City College of San Francisco, Hotel and 
Restaurant Management Program 

Foothill College, Lcs Altos Hills, California 
University of Pittsburgh, SchoQl of Dental 
Medicine 

Tarrant Junior College, Hurst, Texas 
DeAnza College, Cupertino, California 

Solano Community College, Suisun, California 
Clarrisa School, Pittsburgh, Pennsylvania 

Sequoia High School, Redwood City, California 



Penn Technical Institute, Pittsburgh, 

Pennsylvania 
Monterey Peninsula College, Monterey, 

California 

Palo Alto Treatment Facility, California 



Carpenter San Jose Regional Occupational Center 

(partially) 

Laney Community College, Oakland, California 
Mercer County Area Voc/Tech Schools, Trenton, 
New Jersey 

Diesel Mechanic Hayvard Unified School District (Regional 

Occupational Center), Hayward, California 
Mercer County Area Voc/Tech Schools, Trenton, 
New Jersey 
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APPENDIX C 



Work Habits Inventory: Assessment of Student 1 
Work Habits 

Work Habits Inventory Mean Importance Ratings 
of Items 



ASSESSMENT OF SrCDEST'S WORK KA3IT3 j 



Student's Mane 



DIRECTIONS: Please indicate the extent co which eacn of the descriptions btlow applies to chis scudenc. 
space 1, 2, 3, or 4, as shown. 

1. * Mark answer space 1 if she description alaosc never applies co chis scudenc. 

2. Mark answer space 2 if che descripcion occasionally applies co chis scuaenc. 

3. Mark answer space 3 if che descripcion frequencly applies co chis student. 

4. Mark answer space 4 if che descripcion almost always, applies cp chis studenc. 



Mark answer 



Circle one number 
in each row 



Circle one numoer 
in each row 



Alaosc 
Never 



1 

1 

1 
1 
1 



AlmOSC 

Always 

3 4 

3 4 

3 4 

3 4 
3 4 
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3 
3 

3 
3 

3 

3 

3 
3 

3 

3 

3 

3 

3 

3 

3 
3 
3 
3 



Alaosc 
Never 



1. Performs dudes effectively vith 
a minimum of supervision. 

2. Is able co work with people of 
various backgrounds* 

3. Maintains self concrol in 
difficulc situacions. 

4. Doesn'c make fun of ochers. 

5. Works wichouc being easily 
| distracted. 

{ 6. Organizes work to make best 
i use of tine. 

| 7. Is tactful when pointing out 
i mistakes of others, 
t S. Avoias "bacsoucniag" the school. 

j 9. Does acceptable work in che 
j absence cf supervision. 

j 10. -Accomplishes casks as scheduled. 

» ii. Does noc cake equipttenc from 
! ochers without permission. 

j 12. Is cheerful and friendly toward 
| other students. 

13. Meets performance expectations , 
even if these require extra effort 
beyond what is normally expected. 

14. Follows instructions willingly. 

15. Asks questions or asks for help 
when needed. 

16. Wears clothes that are consistent 
with school requirements. 

17. Works well in a team, when 
required, 

13. Does not waste materials or 
supplies. 

19. Communicates effectively to 
fellow students and teacher(s) . 

20. Gets work done, despite 
difficulties. 

21. Uses slack time productively 
without waiting for directions 
from teacher. 

22. Sticks with a task until it is 
finished. 

23. Accepts responsibility for own 
errors. 

24. Shows cooperation and consider- 
ation in working with others. 

25. Takes good care of tools and 
equipment* 



6& 



Almost 
Always 

4 



,1 2 


3 


1 

4 I 

i 


1 2 


3 


t 

4 1 


1 2 


3 


4 


1 2 


3 


4 


1 2 


3 


4 


1 2 


3 


4 


1 2 


3 


4 

1 


i 2 


3 


4 


1 2 


3 


4 


1 2 


3 


4 


1 2 


3 


4 


1 2 


3 


4 


1 2 


3 


4 


x *» 


3 


4 


1 2 


3 


4 



26. 

27. 

28. 

29. 
30. 

31. 

32. 

33. 
34. 

35. 
36. 
37. 
38. 
39. 
40. 
41. 
42. 
43. 
44. 



Shows up at school ready to work. 

Seeks information about respon- 
sibilities when in doubt. 

Makes independent decisions, 
when necessary. 

Shows patience when teaching ochers. 
Does noc disc race or annoy .ochers. 

Comes co school on che days 
scheduled. 

Is eager co I earn about new 
developments and ways of doing 
things. 

Does fair share of che work. 

Adapcs work ha': its to changes in 
rules, procedures, and situacions. 

Is careful noc co endanger che 
safety of ochers. 

Shows a friendly incerasc in 
QCher people. 

Follows' appropriate school safety 
regulations . 

Demonscraces a posicive accicude - 
coward school. 

Gees work done on cine. 

Sets a good example for ocher 
students. 

Follows school" piles abouc 
ascendance, cardiness, etc. 

Maintains personal appearance chac 
is suitable for che school. 

Comes co school on cime. 



Makes an efforc co meec ochers half 
way in conflict situations. 

45. Accepcs responsibility for cask and 
follows chrough wich a minimum 

of supervision* 

46. Volunceers to help when chere is 
a crisis or heavy work load. 

47. Nocifies appropriate individuals 
if unable co come to school as 
scheduled. 

48. Responds well co suggescions 
for improvemenC. 

49. ' Is able co adjusc co new and 

differenc work situacions. 



Work Habits Inventory 
Mean Importance Ratings of Items 

(N = Number of Reviewers)- J *; 



t /// // 4? // / / // / /; / t% i // // * 



1 3.59 3.77 3.58 3.72 3.67 3.60 3.51 3.61 3.7* 3.86 3.60 3.46 3.68 3.74 3.70 3.50 3.55 

2 2.91 3.05 3.08 3.30 3.87 3.80 3.46 3.46 3.57 ■ 3.63 3.00 3.54 3.26 2.94 3.11 3.29 3.07 

3 3.56 3.65 3.54 3.50 3.80 3.73 3.72 3.02 3.8* 3.98 3.60 3.75 3.51 3.61 3.50 3.29 3.56 

4 2.91 3.27 2.8! 3.37 3.67 3.79 3.25 3.26 3.49 3.64 3.60 3.38 3.05 2.71 2.73 2.97 2.93 

5 3.21 3.45 3.31 3.60 3.33 3.60 3.41 3.21 3.37 3.41 3.60 3.25 3.33 3.03 3.11 3.38 3.41 

6 3.52 3.64 3.42 3.70 3.53 3.67 3.64 3.50 3.67 3.76 3.60 3.54 3.57 3.32 3.39 ■ 3.59 3.71 

7 3.26 3.50 3.15 3.65 3.40 3.53 3.38 3.46 3.52 3.57 3.60 3.25 3.30 3.39 3.05 3.12 3.13 
d 3.38 3.77 3.35 3.47 3.60 3.73 3.71 3:71 3.73 3.60 3.40 3.79 3.53 3.26 3.05 3.53 3.57 
9 3.62 3.73 3.77 3.74 3.73 3.73 3.55 3.59 3.55 3.90 4.00 3.46 3.73 3.77 3.73 3.56 3.70 

10 3.65 3.45 3.77 3.77 3.40 3.47 3.67 3.50 3.51 3.76 3.80 3.75 3.49' 3.58 3.63 3.74 3.52 

11 3.15 3.41 *3.12 3.24 2.67 3.07 3.09 2.02 2.95 3.17 3.60 3.13 3.42 3.00 3.09 3.56 3.56 

12 3.21 3.32 2.88 3.26 3.67 3.60 3.46 3.46 3.63 3.40 2.80 3.50 2.99 2.94 2.89 3.09 3.04 

13 3.47 3.18 3.38 3.58 3.53 3.47 3.36 3.43 3.38 3.44 3.40 3.21 3.26 3.26 3.23 3.29 3.45 

14 3.65 3.86 3.69 3.77 3.73 3.67 3.75 3.71 3.74 3.74 3.60 3.63 3.51 3.68 3.55 3.71 3.59 

15 3.79 3.64 3.85 3.84 3.60 3.80 3.62 3.86 3.55 3.90 3.60 3.46 3.72 3.68 3.61 3.65 3.73 

16 3.24 3.14 2.73 2.80 3.53 3.67 3.53 3.61 3.39 3.44 2.60 3.75' 3.04 2.84 2. 04 3.38 3.23 

17 3.32 3.45 3.50 3.53 3.40 3.73 3.52 3.68 3.73 3. 81 3.40 3.67 3.42 3.23 3.50 3.53 3.52 

18 3.47 3.59 3.23 3.26* 3.40 3.33 3.63 3.21 3.39 3.24 3.60 3.67 3.48 3. 16 3.39 3.53 3.57 

19 3.53 3.64 3.58 3.67 3.60 3.73 3.53 3.66 3.60 3.03 3.40, 3.50 3.53 3.55 3.59 3.29 3.43 

20 3.35 3.55 3.48 3.63 3.60 3.60 3.57 3.61 3.54 3.69 3.80 3.46 3.58 3.58 3.43 3.44 3.56 

21 3.47 3.50 3.42 3.44 3.73 3.67 3.53 3.61 3.48 . 3.40 4.00 3.38 3.51 3.23 3.30 3.47 3.45 

22 3.50 3.59 3.65 3.62 3.33 3.60 3.55 3.66 3.55 3.73 3.80 3.67 3.71 3.35 3.36 3.74 3.69 

23 3.53 3.59 3.69 3.70 3.47 3.80 3.57 3.75 3.56 3.83 3.00 3.54 3.68 3.52 3.61 3.59 3.70 

24 3.32 3.36 3.35 3.65 3.60 3.47 3.53 3.61 3.68 3.70 3.60 3.56 3.42 3.26 3.41 ' 3.53 3.50 

25 3.71 3.77 3.65 3.72, 3.53 3.40 3.53 3.24 3.74 3.60 3. SO 3.42 3.72 3.32 3.64 3.59 3.72 
26' 3.79 3.86 3.77 3.77 3.87 3.87 3.92 3.63 3.83 3.74 3.80 3.92 3.68 3.84 3.82 3.94 3.89 

27 3.56 3.45 3.50 3.60 3.53 3.67 3.66 3.69 3.57 4.00 3.40 3.42 3.38 3.*0 3.61 3.47 3.56 

28 3.24 3.36 3.23 3.42 3.07 3.27 3.12 3.48 3.31 3.48 3.60 3.08 3.59 3.58 3.43 3.29 3.39 

29 2.82 3.18 3.12 3.30 3.27 3.40 3.30 3.31 3.36 3.69 3.60 3.25 3.31 3.13 3.05 3.00 3.14 

30 2.97 3.50 3.42 3.56 3.53 3.53 3.47 3.28 3.43 3.48 3.20 3.08 3.41 3.23 3.07 3.35 3.34 

31 3.70 3.82 3.92 3.84 3.67 3.87 3.92 3.79 3.89 3.06 3.60 3.92 3..84 3.81 3.89 3.94 3.09 

32 3.50 3.64 3.36 3.40 3.40 3.40 3.34 3.34 3.56 3.43 3.40 3.29 3.44 3.03 3.21 3.32 3.55 

33 3.35 3.59 3.65 3.72 3.73 3.60 3.62 3.55 3.68 3.64 3.60 3.63 3.64 3.53 3.59 3l74 3.65 

34 3.32 3.36 3.50 3.60 3.67 3.80 3.58 3.66 3.51 3.60 3.40 3.54 3.43 3.48 3.55 3.56 3,56 

35 3.79 3.95 3.65 3.44 3.47 3.67 3.70 3.62 3.69 .3.98 3.80 3.03 3.79 3.04 3.90 3.91 3.91 

36 2.74 3.14 2.69 2.06 3.60 3.60 3.36 3.52 3.51 3.24 2.60 3.46 2.81 2.71 2.77 2.79 2.95 

37 3.88 3.62 3.73 3.44 3.40 3.60 3.63 3.45 3.62 3.98 3.80 3.71 3.68 3.64 3.95 3.94 3.06 

38 3.52 3.77 3.25 3.37 3.67 1.67 3.67 3.62 3.75 3.44 3.40 3.56 3.43 3.19 3.16 3.56 3.65 

39 3.68 3.41 3.77 3.06 3.40 3.40 3.68 3. 52 3.49 3.52 3.60 3.67 3.53 3.48 3.64 3.65 3.60 

40 3.24 3.32 3.23 3.33 3.47 3.67 3.51 3.36 3.43 3.33 3.40 3.43 3.20 3.16 3.09 3.35 3.40 

41 3.42 3.64 3.58 3.50 3.80 3.73 3.63 3.52 3.64 3.67 3.40 3.58 3.60 3.43 3.70 3.53 3.70 

42 2.97 3.36 2.88 2,93 3.67 3.80 3.57 3.52 3.66 3.43 3.20 3.79 3.03 2.97 2.73 3.06 3.14 

43 3.68 3.73 3.77 3.74 3.93 3.80 3.95 3.86 3.62 3.67 3.60 3.88 3.64 3.71 3.06 3.97 3.06 

44 3.06 3.36 3.15 3.35 3.60 3.53 3.20 3.36 3.46 3.40 3.60 3.00 3.22 3.32 3.18 " 3.09 3.18 

45 3.47 3.73 3.65 3.70 3.47 3.67 3.54 3.72 3.60 3.90 3.80 3.50 3.65 3.01 3.66 3.44 3.60 

46 3.24 3.32 3.36 3.51 3.33 3.47 3.28 3.46 3.59 3.60 3.80 3.25 3.29 3.16 3.25 3.38 3.18 

47 3.70 3.68 3.88 3.77 *.87 3.93 3.92 3.06 3.84 3.93 3.80 3.88 3.77 3.77 3.02 3.97 3. OS 

48 3.29 3.50 3.36 3.44 3.67 3.73 3.49 3.41 3.64 3.55 3.60 3.42 3.47 3.32 3.23 3.29 3.39 

49 3.24 3.36 3.54 3.56 3.60 3.73 3.45 3.55 3.55 3.60 3.60 3.36 3.49 3.52 3.33 3.35 3.42 



Journal Articles and Newsletters 

Articles about the Vocational Competency Measures Project appeared in the 
following journals and newsletters: 

• VocEd (Journal of the American Vocational Association) 

• School Shop 

• JcflA nal of the American Personnel and Guidance A ssociation 
^ 

• Journal of Industry-Education Cooperation 

• . Journal of the American Technical Education Association 

• Center Critiques (The East Central Network for Curriculum Coordination) 

• CAPTRENDS 

• Journal of the American Physical Therapy Association 

i 

• Education Daily 

• Report on Education Research 

• Vocational Education and ManPower Weekly 

• Education and Work 

• Update (American Vocational Association) 

• Guidepost (American Personnel and Guidance Association) 

• News Exchange (Association for Supervision and Curriculum Development) 

• Newsletter (National Vocational Guidance Association) 

• Newsnotes (Association for Measurement and Evaluation in Guidance) 

• Cent erg ram (National Center for Research in Vocational Education) 

• Career Education News 

• National Report for Training, and Development (American Society for 
Training and Development) 

• Educational Measurement 

• Data Processing Digest 

• NATTS News (National Association of Trade and Technical Schools) 

• Newsletter (American Apparel Manufacturing Association) 



Industrial Relations Bulletin (American Electronics Association) 
Open Entries (The Center for Studies in Vocational Education) 
Newsletter (Niagara Frontier Industry Education Council, Inc.) 
Implement and Tractor 
American Fabrics and Fashion Magazine 

• Newsletter (Greater Providence Chamber of Commerce) 

• Newsletter (Phi Delta Kappa) 

• Electronic Servicing and Technology 

• Newsletter (American Association of Women Dentists) 

• Career Planning and Adult Development Newsletter 

• Newsletter (National Association of Industry-Education Councils) 

• Restaurant Hospitality 

• Newsletter (Wayne County Intermediate School District) 

• Apparel Industry Magazine 

• Linkages (National Institute for Staff and Organizational Development) 



Presentations to Selected Professional Organizations 

Presentations about the Vocational Competency Measures project Were made t 
the following professional organizations: 

• California Association of Program Evaluators, March 1980 

• American Personnel and Cuidance Association, April 1981 

• California Water Pollution Control Association, November 1981 

• Association of Health Career Schools, January 1982 

• Fourth International Learning Congress of the Society for Applied 
^Learning Technology, February 1982 

e* . 
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• Michigan Association for Measurement and Evaluation In Guidance, 

March 1982 

• Eighth Annual Convention of the Vocational Education Association of New 
Jersey, March 1982 

• California Community Colleges, March 1982 

• Wisconsin Vocational Education Association, March 1982 (materials 
displayed) 

• AVA Region V Leadership Conference, March 1982 (materials displayed) 

• American Association of Community and Junior Colleges, April 1982 

• American Technical Education Association, April 1982 

• California Community Colleges Gender Equity Conference for Vocational 
Education, April 1982 

• New Jersey Vocational Education Association, April 1982 

• California Association of Vocational Educators, April 1982 

• Business Education Association of Metropolitan New York, May 1982 

• West Virginia Vocational Administrators, May 1982 

• National Vocational Home Economics Association, May 1982 

• American Physical Therapy Association, June 1982 

• California Association of Health Careers Educators, June 1982 

• L Michigan Occupational Education Association, August 1982 

• West Virginia Vocational Teachers Association, August 1982 

• Alabama Association of Secondary Vocational Educators, August 1982 

• International Union of Operating Engineers, September 1982 

• Military Interservice Review Organization, September 1982 

• Idaho Vocational Educators Association, November 1982 
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